Methods and Systems for Storing and Retrieving Data

ABSTRACT

Through use of the technologies of the present invention, one is able to store and to retrieve data efficiently. One may realize these efficiencies by coding the data and storing coded data that is of a smaller size than original data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. Ser. No. 13/756,921,which was filed on Feb. 1, 2013 and is also a continuation-in-part ofU.S. Ser. No. 13/797,093, which was filed on Mar. 12, 2013. The entiredisclosure of each of the afore-referenced priority applications isincorporated by reference into the instant specification.

FIELD OF THE INVENTION

The present invention relates to the field of data storage andretrieval.

BACKGROUND OF THE INVENTION

The twenty-first century has witnessed an exponential growth in theamount of digitized information that people and companies generate andstore. This information is composed of electronic data that is typicallystored on magnetic surfaces such as disks, which contain small regionsthat are sub-micrometer in size and are capable of storing individualbinary pieces of information.

Because of the large amount of data that many entities generate, thedata storage industry has turned to network-based storage systems. Thesetypes of storage systems may include at least one storage server thatforms or is part of a processing system that is configured to store andto retrieve data on behalf of one or more entities. The data may bestored and retrieved as storage objects, such as blocks and/or files.

One system that is used for storage is a Network Attached Storage (NAS)system. In the context of NAS, a storage server operates on behalf ofone or more clients to store and to manage file-level access to data.The files may be stored in a storage system that includes one or morearrays of mass storage devices, such as magnetic or optical disks ortapes. Additionally, this data storage scheme may employ Redundant Arrayof Independent Disks (RAID) technology.

Another system that is used for storage is a Storage Area Network (SAN).In a SAN system, typically a storage server provides clients withblock-level access to stored data, rather than file-level access to it.However, some storage servers are capable of providing clients with bothfile-level access and block-level access.

Regardless of whether one uses NAS or SAN, all industries that generatedata must consider the cost of storing and retrieving that data.Therefore, there is a need for new technologies for economically storingand retrieving data.

SUMMARY OF THE INVENTION

The present invention provides methods, systems and computer programproducts for improving the efficiency of storing and retrieving data. Byusing various embodiments of the present invention, one can efficientlystore and access data that optionally has been converted or encoded forsecurity. Because the present invention separates metadata from rawdata, there is no limitation based on the type of file that can bestored and/or retrieved in connection with this invention. Additionally,through the various embodiments of the present invention one maytransform data and/or change the physical devices on which thetransformed or converted data is stored. This may be accomplishedthrough automated processes that employ a computer that comprises or isoperably coupled to a computer program product that when executedcarries out one or more of the methods or processes of the presentinvention. These methods or processes may for example be embodied in orcomprise a computer algorithm or script and optionally be carried out bya system.

According to a first embodiment, the present invention is directed to amethod for storing data on a non-cache recording medium comprising: (i)receiving an I/O stream of N Bytes (using for example an I/O protocol);(ii) fragmenting the N Bytes into fragmented units of X Bytes; (iii)applying a cryptographic hash function (which is a value algorithm) toeach fragmented unit of X Bytes to form a generated hash function valuefor each fragmented unit of X Bytes; (iv) accessing a correlation file,wherein the correlation file associates a stored hash function value ofY bits with each of a plurality of stored sequences of X Bytes and (a)if the generated hash function value for a fragmented unit of X Bytes isin the correlation file, using the stored hash function value of Y bitsfor storage on a non-cache recording medium; and (b) if the generatedhash function value for the fragmented unit of X Bytes is not in thecorrelation file, then storing the generated hash function value of Ybits with the fragmented unit of X Bytes in the correlation file andusing the generated hash function value for storage on the non-cacherecording medium.

According to a second embodiment, the present invention is directed to amethod for storing data on a non-cache recording medium comprising: (i)receiving an I/O stream of N Bytes; (ii) fragmenting the N Bytes intofragmented units of X Bytes; (iii) associating a cryptographic hashfunction value with each fragmented unit of X Bytes, wherein saidassociating comprises accessing a correlation file, wherein thecorrelation file associates a stored hash function value of Y bits witheach of a plurality of stored sequences of X Bytes and (a) if thesequence of fragmented unit of X Bytes is in the correlation file, usingthe stored hash function value of Y bits for storage on a non-cacherecording medium; and (b) if the sequence of fragmented unit of X Bytesis not in the correlation file, then storing a new generated hashfunction value of Y bits with the fragmented unit of X Bytes in thecorrelation file and using the new generated hash function value of Ybits for storage on the non-cache recording medium.

According to a third embodiment, the present invention is directed to amethod for storing data on a non-cache recording medium comprising: (i)receiving an I/O stream of N Bytes; (ii) applying a cryptographic hashfunction to each unit of N Bytes to form a generated hash function valuefor each unit of N Bytes; (iii) accessing a correlation file, whereinthe correlation file associates a stored hash function value of Y bitswith each of a plurality of stored sequences of N Bytes and (a) if thegenerated hash function value for the unit of N Bytes is in thecorrelation file, using the stored hash function value of Y bits forstorage on a non-cache recording medium; and (b) if the generated hashfunction value for the unit of N Bytes is not in the correlation file,then storing the generated hash function value of Y bits with the unitof N Bytes in the correlation file and using the generated hash functionvalue for storage on the non-cache recording medium.

The various methods of the present invention may for example be used inconnection with the following method for configuring storage systems tostore electronic data: (i) receiving a set of parameters, wherein theparameters comprise one or more, if not all of file system information,bootability information and partition information; (ii) receivingmetadata; and (iii) storing the parameters and metadata on a mediator.After configuration of the storage system, the system may be ready toreceive one or more files, wherein each file has a file name; to storeeach of the files (optionally as processed through one of theaforementioned methods) on a non-cache medium at a location; and tostore on the mediator, a correlation of each file name with a locationon the non-cache medium.

When employing the various methods of the present invention,instructions may be stored on a computer program product that is encodedon a non-transitory computer-readable medium, operable to cause a dataprocessing apparatus to perform operations comprising: (a) receiving,from a server, an I/O stream of N Bytes through an I/O protocol; (b)obtaining a hash function value for each unit of N Bytes or for aplurality of fragmented units of N Bytes; and (c) causing data to bestored that comprises either a plurality of hash values or a pluralityof converted hash values. By converting the hash values into convertedhash values, there may be heightened protection against unauthorizedaccess to information within a stored file. Optionally, the computerprogram product further comprises a conflict resolution module thatpermits identification of the same hash function value being assigned todifferent units of N Bytes or different fragmented units of N Bytes, andcauses different data to be stored that corresponds to each of theconflicting hash function values. The computer program products, whenexecuted, can cause initiation and automatic execution of theaforementioned features. These instructions may be stored in one moduleor in a plurality of separate modules that are operably coupled to oneanother.

Various embodiments of the present invention may also be implemented onsystems. An example of a system of the present invention comprises: (a)a non-cache storage medium (which may be a persistent or non-transitorystorage device) and (b) one or more processors operable to interact withthe non-cache storage medium. Optionally, the system further comprisesone or more user interfaces (e.g., graphic use interfaces) that arecapable of allowing a user to interact with the one or more processors.The one or more processors are further operable to carry out one or moreof the methods of the present invention, which may for example be storedon a computer program product that is stored in a non-transitory medium.

In some embodiments, the system further comprises: (i) a mediator,wherein the mediator is stored remotely from the non-cache data storagemedium, and the mediator comprises: (a) a first set of tracks; (b) asecond set of tracks; (c) a third set of tracks; and (d) a fourth set oftracks; (ii) a non-cache medium; and (iii) a manager, wherein themanager is configured: (a) to cause storage of data comprising one ormore of file system information, bootability information and partitioninformation in the first set of tracks; (b) to cause storage of metadatain the third set of tracks; (c) to cause storage of one or more files onthe non-cache medium, wherein the one or more files are stored on thenon-cache medium without any of file system information, bootabilityinformation and partition information; (d) to cause storage in thefourth set of tracks of the location of each file in the non-cachemedium; and (e) to cause storage of a correlation of the location ofeach file in the non-cache medium with a host name for a file.

Through the various embodiments of the present invention, one canincrease the efficiency of storing and retrieving data. The increasedefficiency may be realized by using less storage space than is used incommonly used methods and investing less time and effort in the activityof storing information. In some embodiments, one can also increaseprotection against unauthorized retrieval of data files. These benefitsmay be realized when storing data either remotely or locally, and thevarious embodiments of the present invention may be used in conjunctionwith or independent of RAID technologies.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a representation of a mediator and non-cache medium (NCM).

FIG. 2 is a representation of a system for storing information using amediator.

FIG. 3 is a representation of a system for using two mediators to backup information that is stored.

FIG. 4 is a representation of a binary tree.

DETAILED DESCRIPTION OF THE INVENTION

Reference will now be made in detail to various embodiments of thepresent invention, examples of which are illustrated in the accompanyingfigures. In the following detailed description, numerous specificdetails are set forth in order to provide a thorough understanding ofthe present invention. However, unless otherwise indicated or implicitfrom context, the details are intended to be examples and should not bedeemed to limit the scope of the invention in any way.

DEFINITIONS

Unless otherwise stated or implicit from context the following terms andphrases have the meanings provided below.

The term “bit” refers to a binary digit. It can have one of two values,which can be represented by either 0 or 1. A bit is the smallest unitthat is stored on a recording medium.

The term “block” refers to a sequence of bytes or bits of data having apredetermined length.

The phrases “bootability code,” “bootability information” and“bootability feature” refer to information that provides the means bywhich to enter a bootable state and may be stored on a boot sector. Aboot sector may contain machine code that is configured to be loadedinto RAM (random access memory) by firmware, which in turn allows theboot process to load a program from or onto a storage device. By way ofexample, a master boot record may contain code that locates an activepartition and invokes a volume boot record, which may contain code toload and to invoke an operating system or other standalone program.

The term “byte” refers to a sequence of eight bits.

The term “cache” refers to the location in which data is temporarilystored in order for future requests for the data to be served faster orfor the purposes of buffering. The L1 cache (level 1 cache) refers to astatic memory that is, for example, integrated with a processor core.The L1 cache may be used to improve data access speed in cases in whichthe CPU (central processing unit) accesses the same data multiple times.The L2 cache (level 2 cache) is typically larger than the L1 cache, andif a data file is sought but not found in a L1 cache, a search may bemade of a L2 cache prior to looking to external memory. In someembodiments, the L1 cache is not within a central processing unit.Instead, it may be located within a DDR, DIMM or DRAM. Additionally oralternatively, L2 cache may be part of PCI2.0/3.0, which goes into amotherboard. Thus, each of L1 cache and L2 cache may be in separateparts of a motherboard. With respect to size, in some embodiments of thepresent invention L1 cache is between 2 gigabytes and 128 terabytes orbetween 2 gigabytes and 4 terabytes; and L2 cache is between 16gigabytes and 1 petabyte or between 16 gigabytes and 3.2 terabytes.

The term “chunklet” refers to a set of bits that may correspond to asector cluster. The size of a chunklet is determined by the storagesystem and may have a chunklet size. Traditionally, the chunklet sizewas derived by the CHS scheme, which addressed blocks by means of atuple that defines the cylinder, head and sector at which they appearedon hard disks. More recently, the chunklet size has been derived fromthe LBA measurement, which refers to logical block addressing, and isanother means for specifying the location of blocks of data that arestored on computer storage devices. By way of example, the chunklet sizemay be 512B, 1K, 2K, 4K, 8K, 16K, 32K, 64K or 1 MB. As persons ofordinary skill in the art are aware 1K=1024B. Chunklets may be receivedas raw data from a host.

The term “cubelet” refers to a virtual portion of capacity of a storagemedium. Thus, it is a three dimensional space within what may be anon-cache medium. Within a cubelet, binary data that corresponds to afile or to part of a file may be stored, and from the cubelet this datamay be retrieved. Thus, a cubelet is a virtual measurement and isanalogous to a LUN (logic unit number).

A “file” is a collection of related bytes or bits having an arbitrarylength. A file may be smaller than a chunklet, the same size as achunklet or larger than a chunklet.

The phrase “file name” refers to a notation or code that permits acomputer to identify a specific file and to distinguish that file fromother files.

The phrase “file system” refers to an abstraction that is used to store,to retrieve and to update a set of files. Thus, the file system is thetool that is used to manage access to the data and the metadata offiles, as well as the available space on the storage devices thatcontain the data. Some file systems may for example reside on a server.Examples of file systems include but are not limited to the Unix filesystem and its associated directory tables and Modes, Windows FAT16 andFAT32 file systems (FAT refers to File Allocation Table), Window NTFS,which is based on master file tables, and Apple Mac OSX, which uses HFSor HFS plus.

The phrases “hash function,” “cryptographic hash function valuealgorithm” and “hash function value algorithm” refer to an algorithm orsubroutine that maps large data sets (optionally of variable length) tosmaller data sets that have a fixed length for a particular hashfunction. A “hash function value” refers to the output that is returnedafter application of a hash function algorithm. The values that thealgorithm returns may also be called hash values, hash codes, hash sums,checksums or hashes. When, for example, using MD5, the output is 128bits, whereas when using SHA-1, the output is 256 bits.

The terms “host” and “initiator” may be used interchangeably and referto the entity or system that sends data for storage to the data storageand retrieval mediation system of the present invention. The host maysend data that corresponds to one or more types of documents or filesand received data. Preferably, within any input/output (“I/O”) stream,the data corresponds to a file of a single document type.

The abbreviation “LBA” refer to logical block addressing. LBA is alinear addressing scheme and is a system that is used for specifying thelocation of blocks of data that are stored in certain storage media,e.g., hard disks. In a LBA scheme, blocks are located by integer numbersand only one number is used to address data. Typically, the first blockis block 0.

The term “manager” refers to a computer program product, e.g., code thatmay be stored in a non-transitory medium and that causes one or moreother actions to be taken, e.g., receiving, transmitting or processingdata. It may be stored on hardware, software or a combination thereof.In some embodiments, the manager may be part of a computer and/or systemthat is configured to permit the manager to carry out its intendedfunction.

The term “mediator” refers to a computer program product that may bestored on hardware, software or a combination thereof, and thatcorrelates one or more units of storage space within at least onenon-cache medium with a file name. A mediator may be orders of magnitudesmaller than the non-cache medium to which it points. For example, itmay be approximately as small as about 0.2% of the size of a typicalcylinder. In some embodiments, the mediator may exist in a computingcloud, whereas in other embodiments, it exists in a non-transitorytangible recording medium. The mediator may be able to organize, totranslate and to control the storage of data in locations that hostsperceive as being in certain tracks of recording media while actuallyoccurring in different tracks of recording media or it may be operablycoupled to a manager that serves one or more if not all of thesefunctions. Furthermore, the mediator may comprise a sector map, a tableor other organization of data that may be located within a physicaldevice or structure, and thus the contents of the mediator may cause thephysical device or structure to have certain geometry.

The term “metadata” refers to the administration information aboutcontainers of data. Examples of metadata include but are not limited tothe length or byte count of files that are being read; informationpertaining to the last time files were modified; information thatdescribes file types and access permissions; and cubelet QoS, VM andWORM. Other types of metadata include operating system information,auto-initialization information, group permissions, and frequency ofbits within the document type. In some embodiments, stored metadata mayfor example be used to permit efficient contraction or expansion ofstorage space for an initiator as the number and size of documents thatit seeks to store shrinks or grows.

The abbreviation “NAS” refers to network area storage. In a NAS system,a disk array may be connected to a controller that gives access to alocal area network transport.

The phrase “operably coupled” means that systems, devices and/or modulesare configured to communicate with each other or one another and areable to carry out their intended purposes when in communication or afterhaving communicated.

The phrase “operating system” refers to the software that managescomputer hardware resources. Examples of operating systems include butare not limited to Microsoft Windows, Linux, and Mac OS X.

The term “partition” refers to formats that divide a storage medium,e.g., a disk drive into units. Thus, the partition may also be referredto as a disk partition. Examples of partitions include but are notlimited to a GUID partition table and an Apple partition map.

The abbreviation “RAID” refers to a redundant array of independentdisks. To the relevant server, this group of disks may look like asingle volume. RAID technologies improve performance by pulling a singlestrip of data from multiple disks and are built on one or multiplepremise types such as: (1) mirroring of data; (2) striping data, or (3)a combination of mirroring and striping of data.

The phrase “recording medium” refers to a non-transitory tangiblecomputer readable storage medium in which one can store magnetic signalsthat correspond to bits. By way of example, a recording medium includesbut is not limited to a non-cache medium such as hard disks and solidstate drives. As persons of ordinary skill in the art know, solid statedrives also have cache and do not need to spin. Examples ofnon-transitory tangible computer readable storage medium include, butare not limited to, a hard drive, a hard disk, a floppy disk, a computertape, ROM, EEPROM, nonvolatile RAM, CD-ROM and a punch card.

The abbreviation “SAN” refers to a storage area network. This type ofnetwork can be used to link computing devices to disks, tape arrays andother recording media. Data may for example be transmitted over a SAN inthe form of blocks.

The abbreviation “SAP” refers to a system assist processor, which is anI/O (input/output) engine that is used by operating systems.

The abbreviation “SCSI” refers to a small computer systems interface.

The term “sector” refers to a subdivision of a track on a disk, forexample a magnetic disk. Each sector stores a fixed amount of data.Common sector sizes for disks are 512 bytes (512B), 2048 bytes (2048B),and 4096 bytes (4K). If a chunklet is 4K in size and each sector is 512Bin size, then each chunklet corresponds to 8 sectors (4*1024/512=8).Sectors have tracks and are located on platters. Commonly, two or fourplatters make up a cylinder, and 255 cylinders make up hard disk andmedia devices.

The phrase “sector map” refers to the tool that receives calls from ahost and correlates locations in a storage device where a file isstored. A sector map may, for example, operate under parameters that aredefined by a SCSI protocol. In some embodiments of the presentinvention, the sector map may be located in a bit field of a mediator.

The term “track” refers to a circular unit within a disk thattransverses all sectors. A “track sector” is a track within any onesector. A “track cluster” spans more than one sector.

Preferred Embodiments

The present invention provides methods for storing data on a non-cacherecording media, computer program products for carrying out thesemethods, and systems that are configured to carry out these methods.Through various embodiments of the present invention, one canefficiently store and retrieve data.

According to one embodiment, the present invention provides a method inwhich a stream of N Bytes is received. The stream may be received over anetwork that is wired or wireless through known methods and technologiesfor transmitting I/O streams, and the stream may be sent by a host.

After receipt of the stream of data, in some embodiments the methodcalls for fragmenting the N Bytes into fragmented units of X Bytes. Byfragmenting an I/O stream, the method divides each I/O stream intosmaller units (which also may be referred to as blocks) of predeterminedand preferably uniform size. As persons of ordinary skill in the artwill recognize one benefit of using smaller block sizes is that smallblocks allow for easier processing of data. By way of a non-limitingexample, an I/O stream may be 4096 Bytes long, and the method mayfragment that stream into fragmented units that are 1024, 512, 256 or128 Bytes long.

Next a cryptographic hash function value algorithm may be applied toeach fragmented unit of X Bytes to form a generated hash function valuefor each fragmented unit of X Bytes. Examples of hash function valuealgorithms include, but are not limited to, MD5 hash (also referred toas a message digest algorithm), MD4 hash and SHA-1. The value that isoutput from a hash function value algorithm may be referred to as achecksum or sum. In some examples, the sum is 64, 128 or 256 bits insize. Because of the highly repetitive nature of data within I/Ostreams, the probability of generating conflicting sums, i.e. sums thatare the same but correspond to different fragmented units, is relativelylow.

After a checksum is obtained for each fragmented unit, a correlationfile is accessed. The method may obtain checksums according to a firstin first out (“FIFO”) protocol and either begins accessing thecorrelation file while the I/O streams are being received, whilechecksums are being generated, or after all I/O streams have beenreceived, fragmented and subjected to the hash function value algorithm.The correlation file correlates a different stored hash function valueof Y bits with each of a plurality of stored sequences of X Bytes and(a) if the generated hash function value for a fragmented unit of XBytes is in the correlation file, the method causes the stored hashfunction value of Y bits to be used for storage on a non-cache recordingmedium; and (b) if the generated hash function value for the fragmentedunit of X Bytes is not in the correlation file, the method causes thegenerated hash function value of Y bits to be stored with the fragmentedsequence of X Bytes in the correlation file and uses the generated hashfunction value for storage on the non-cache recording medium. This hashfunction value, itself, may be stored or further processed prior tostorage on the non-cache recording medium.

As noted above, the probability of conflicting hash values beinggenerated as a result of application of the hash value algorithm is low.However, in various embodiments, the present invention also applies aconflict resolution module in order to address circumstances in which aconflict might arise. In these embodiments, there is a conflict modulethat is activated whenever for a fragmented unit of X Bytes, a hashvalue is generated that is the same as a hash value in a correlationfile. This module then queries whether the fragmented unit of X Bytes isthe same as the already stored value of X Bytes for that hash value. Ifthe conflict resolution module determines that for a fragmented unit ofX Bytes, a hash function value is generated that is the same as a storedhash function value in the correlation file, but the fragmented unit ofX Bytes is different from the X Bytes associated with the stored hashfunction value, then the module causes there to be different Z bitsassociated with the stored hash function value and the generated hashfunction value.

In some embodiments, in the correlation file, for all hash functionvalues for which no conflict exists, Z bits are associated and the Zbits are a uniform length of e.g., 8 to 16 zeroes. By way of anon-limiting example N=4096, X=512, Y=128 or 256, and Z=8 or 16. Themethod may associate 8 zeroes at the end of a checksum when the checksumdoes not conflict with a previously stored checksum. Upon identificationof a conflict, (e.g., different fragmented units being associated withthe same checksum,) the newest checksum may be assigned a different Zvalue. Thus, if the Z value as stored in the correlation file is00000000, the Z value for the first conflicting checksum may be 00000001and should there be another conflicting checksum 00000010. If there wereadditional conflicting checksums, each conflicting checksum may beassigned the next Z value as the conflicting checksum is identified.Thus, the conflict module may be accessed as a check after thecorrelation file is accessed, and only if the newly generated hash valueis already within the correlation file. The conflict module would thendetermine if there is a conflict or if both the checksum and thefragmented units from the received file are already associated with eachother in the correlation file.

Upon obtaining checksums for each fragmented unit, and followingapplication of the conflict module if present, a plurality of hashfunction values may be stored on a non-cache recording medium.

In some embodiments, instead of storing the hash value, the methodfurther comprises converting (also referred to as encoding) each hashfunction value for storage through use of a bit marker table to generatea bit marker for each hash function value for storage and storing eachbit marker for the respective hash function value in the non-cacherecording medium. By using a bit marker table, one may convert or codethe hash values. Thus, the hash values may serve as input when accessingthe bit marker table, and the bit markers may serve as output.

The methods of the present invention may, after obtaining the hashvalues that correspond to the fragmented units of the I/O stream executean algorithm or computer program that is configured to translate thehash values into a set of coded data. This execution may for example becontrolled by a manager that also controls the application of the hashfunction value algorithm. The coded data is also comprised of binarysignals, and it is coded and stored in a manner that permits it to beconverted back into the hash values of the file. Thus, information isretained during the encoding process that permits decoding without aloss of information.

In some embodiments, each hash value is assigned a code that consists ofa plurality of 0s and/or 1s. In other embodiments, each hash value isdivided into a plurality of subunits that are each assigned a code thatconsists of a plurality of 0s and 1s. The subunits may be defined by alength A, wherein the length of the hash value divided by A is aninteger. If any subunit does not have that number of bits, e.g., one ormore subunits have a smaller number of bits than the number within thesubunits that are configured to be converted, the system may add bits,e.g., zeroes, until all subunits are the same size. Alternatively, itmay be performed on the hash values level prior to dividing the hashvalues into subunits.

The encoding can serve two independent purposes. First, by convertingthe data for storage through the bit marker table, there is increasedsecurity. Only a person or entity that knows the code will be able todecode it and to reconstruct the document. Second, if the code iscreated using fewer bits than the hash values that correspond to thedocument, then both the use of converted data and the hash values willcause less storage space to be needed and there can be further costsavings.

In some embodiments, rather than using a bit marker table, the methodfurther comprises encoding each hash function value for storage throughuse of a frequency converter. In these methods, for each hash functionvalue for storage, a converted string of bits is generated, wherein forhash values of the same size, the converted strings of bits that areoutput are of different lengths. The frequency converter may associateeach of a plurality of converted string of bits with each of a pluralityof hash function values, and the plurality of converted string of bitsin the frequency converter are sufficiently distinct that when read backthe converted string of bits can be correlated with the appropriate hashvalues.

To further illustrate how a frequency converter can use strings of bitsof different sizes, reference may be made to FIG. 4. FIG. 4 shows abinary tree for all binary sequences that are five units long and beginwith the number 1. As the tree shows, there are 16 paths to createsequences of five digits in length, beginning at the top row and movingdown the tree to the bottom. However, when moving down the tree, one canmove down a branch to for example the third, fourth or fifth rows.Accordingly, the method may be designed such that for one piece of data,it assigns a code that corresponds to moving partially down the tree,whereas for other pieces of data, the method assigns a code thatcorresponds to moving along a different branch a greater number ofunits. By way of a non-limiting example, for particular pieces ofconverted data, one may stop at the third row, e.g., generating asequence of 101, whereas for all other pieces of data, the first threedigits are not 101. Thus, the methods would never allow use of: 1010,1011, 10100, 10101, 10110 and 10111 (which are within the box in FIG.4). Upon retrieval of data from memory, the system would read digitsuntil it recognizes them as unique, and upon recognition of them asunique, use the unique string as input into a program that allows fordecoding. The various methods of the present invention may, during orafter retrieval call for reading a minimum number of bits and after thatminimum number (which corresponds to the shortest unique code), checkfor uniqueness by comparing to a data file or table, and if the code isnot unique within the data file or table continue to extend the numberof bits and check for uniqueness of the extended code after addition ofeach individual bit until the sequence is recognized as unique. Theaforementioned example is described in terms of strings of 3, 4 or 5bits, but those sizes were used for illustrative purposes only. In someembodiments, the shortest sequences of unique bits are 10 to 15 bitslong and the longest sequences of bits are from 25 to 40 bits long.

In some embodiments, for every plurality of converted strings of bitsthat are of different sizes there is a first converted string of bitsthat is A bits long and a second converted string of bits that is B bitslong, wherein A<B, and the identity of the A bits of the first convertedstring of bits is not the same as the identity of the first A bits ofthe second converted string of bits.

Methods, systems and computer program products for encoding (alsoreferred to as converting) raw data are described in co-pending andcommonly owned patent application entitled Bit Markers and FrequencyConverters, U.S. Ser. No. 13/756,921, filed on Feb. 1, 2013, which isincorporated by reference in its entirety, and are further illustratedin the excerpted tables in the example section of this disclosure.

The bit marker table or frequency converter may be stored within themediator, the manager or elsewhere. However, the bit marker table orfrequency converter is able to be in communication with the mediatorand/or manager. Methods and systems for communicating with files andprograms that are stored locally or remotely are well known to personsof ordinary skill in the art. As Tables I, II and III (see examples)show, information may be converted and the output code can be configuredto take up less space than the input because markers are used torepresent groups of bits. Thus, preferably within a table, at least one,a plurality, at least 50%, at least 60%, at least 70%, at least 80%, atleast 90%, or at least 95% of the markers are smaller in size than thesubunits. However, there is no technological impediment to having theconverted data be the same size or larger than the data received fromthe host or as generated from a hash function value algorithm.

Optionally, after the I/O (output from the host, input to the system) isreceived from a host, the method further comprises acknowledging receiptof the I/O stream to the host. The host may record the file of the I/Ostream as being stored at a first storage address and the convertedstring of bits are stored in the non-cache recording medium at a secondstorage address, wherein the first storage address is not the same asthe second storage address. Further, the first storage address maycorrespond to a first file size and the second storage address maycorrespond to a second file size, wherein the first file size is atleast two times as large or at least four times as large as the secondfile size.

In some embodiments, the second storage address is stored on a mediator.Within the mediator, the second storage address may be stored in a bitfield.

In some embodiments, the mediator comprises: (i) a first set of tracks,wherein the first set of tracks comprises file system information,bootability information and partition information; (ii) a second set oftracks, wherein the second set of tracks comprises a copy of the firstset of tracks; (iii) a third set of tracks, wherein the third set oftracks comprises metadata other than file system information,bootability information and partition information; and (iv) a fourth setof tracks, wherein the fourth set of tracks comprises the bit field.

The aforementioned embodiments describe methods in which each fragment'sunit of bits is subjected to a cryptographic hash function valuealgorithm prior to accessing the correlation file. However,alternatively one could check the correlation file first, and apply acryptographic hash function value algorithm only if the fragmented unitsare not already in the table.

Accordingly, in some embodiments, the present invention provides amethod for storing data on a non-cache recording medium comprising: (i)receiving an I/O stream of N Bytes; (ii) fragmenting the N Bytes intofragmented units of X Bytes; (iii) associating a cryptographic hashfunction value with each fragmented unit of X Bytes, wherein theassociating comprises accessing a correlation file, wherein thecorrelation file correlates a stored hash function value of Y bits witheach of a plurality of stored sequences of X Bytes and (a) if thesequence of fragmented X Bytes is in the correlation file, using thestored hash function value of Y bits for storage on a non-cacherecording medium; and (b) if the sequence of fragmented X Bytes is notin the correlation file, then generating and storing a new hash functionvalue of Y bits with the fragmented sequence of X Bytes in thecorrelation file and using the new hash function value for storage onthe non-cache recording medium.

In this embodiment, after a new hash value is generated, the method maythen check whether there is a conflict, i.e., whether despite therebeing different fragmented units, the algorithm associated the samechecksum with those different fragmented units. If this is the case,then a conflicts module may cause different Z bits to be associated witheach of the conflicting checksums. The number of Z bits may, forexample, be 8-16 or 8 or 16.

The above described methods call for fragmenting the I/O stream of NBytes. However, the methods could be applied by applying a cryptographichash function value algorithm to the I/O stream and not fragmenting it.Accordingly, in some embodiments, the present invention provides amethod for storing data on a non-cache recording medium comprising: (i)receiving an I/O stream of N Bytes; (ii) applying a cryptographic hashfunction value algorithm to each unit of N Bytes to form a generatedhash function value for each unit of N Bytes; (iii) accessing acorrelation file, wherein the correlation file correlates a stored hashfunction value of Y bits with each of a plurality of stored sequences ofN Bytes and (a) if the generated hash function value for the unit of NBytes is in the correlation file, using the stored hash function valueof Y bits for storage on a non-cache recording medium; and (b) if thegenerated hash function value for the unit of N Bytes is not in thecorrelation file, then storing the generated hash function value of Ybits with the fragmented unit of N Bytes in the correlation file andusing the generated hash function value for storage on the non-cacherecording medium.

Alternatively, the method could comprise: (i) receiving an I/O stream ofN Bytes; (ii) associating a cryptographic hash function value with eachunit of N Bytes, wherein said associating comprises accessing acorrelation file, wherein the correlation file correlates a stored hashfunction value of Y bits with each of a plurality of stored sequences ofN Bytes and (a) if the sequence of N Bytes is in the correlation fileusing the stored hash function value of Y bits for storage on anon-cache recording medium; and (b) if the sequence of N Bytes is not inthe correlation file, then generating and storing a new hash functionvalue of Y bits with the N Bytes in the correlation file and using thenew hash function value for storage on the non-cache recording medium.As with other methods, the method can be used with a conflict module.Additionally, although various methods are for illustrative purposesdescribed as being applied to a set of N Bytes or X Bytes, a person ofordinary skill in the art will appreciate that the methods may be andpreferably applied for all N Bytes or X Bytes of a file.

The various method of the present invention may be used in combinationwith other methods that are now known or that come to be known and thata person of ordinary skill in the art would appreciate as being of usein connection with the methods described herein.

The various methods may be controlled automatically by a manager, whichmay comprise one or more modules and reside on a local computer, on anetwork or in a cloud. The manager is configured to coordinate receiptof or to receive certain information itself and to transfer thisinformation to a mediator, or to control receipt of the informationdirectly by the mediator. Thus, the methods can be designed such thatinformation from the initiator flows through the manager to mediator orthat the manager only directs the flow of information to othercomponents of the system.

In various embodiments of the present invention, the manager may applyor cause to be applied, the hash function algorithm, any conflict moduleif present and any conversion module if present and cause the flow ofdata directly to the mediator. The manager also may control storage ofinformation through use of the mediator and retrieval and transmissionof information. When storing the information, an LBA number may beidentified and the data to be stored may be sent to a buffer in order toavoid or to reduce bottlenecking. Further, L1 and/or L2 cache may beused during storage.

Similarly, when retrieving data, the method may call the data from theNCM, fill a buffer, obtain the checksum values, and then recreate thefragmented units and reassemble, or if not fragmented, then directlyreassemble to form information in a form that a host may receive andreview. If the stored data is converted data, prior to obtaining thechecksum values, the data may be decoded.

In some embodiments, a manager may control, communicate with andcoordinate the activities of one or a plurality of mediators. For eachmediator, the manager receives (or coordinates receipt of) a set ofparameters. These parameters may comprise, consist essentially of orconsist of one, two or all three of file system information, bootabilityinformation and partitioning information. The manager causes thisinformation to be stored in a first set of tracks on the mediator, whichmay be referred to as reserve 1 or R₁. The file system will dictate howthe reserve blocks are to be used. For example, when using NTFS, sectors1-2 may be for a MBR (master boot record) and sector 3 may be for $MFT.Optionally, these tracks may be copied into a second set of tracks,which may be referred to as reserve 2 or R₂.

The manager may also receive metadata in addition to the parametersdescribed in the preceding paragraph. The metadata is stored in a thirdset of tracks on the mediator. At the time that the manager receives theparameters and metadata, or at a later time, it may also receive one ormore files for storage on a non-cache medium. Each file is received witha file name. The file name is generated by a host that transmits thefile and may be defined by the host's file system. The manager, whichmay for example be or be a part of a SAN or NAS or combination thereof,upon receipt of the file with a file name, can automatically execute thesteps described herein for storage.

The files of the I/O stream comprise, consist essentially of or consistof a plurality of digital binary signals, e.g., 0s and 1s. These digitalbinary signals of the I/O stream may be organized in chunklets that aredefined by their length, which is an integer number greater than 1.Additionally, within a file, the chunklets have an order. Typically, fora given file, each chunklet contains the same number of bits. If anychunklet does not have that number of bits, e.g., one or more chunkletshas a smaller number of bits the receiving system may add bits, e.g.,zeroes, until all chunklets are the same size.

In certain methods described above, the converted data is formed fromthe hash values. However, it is within the scope of the presentinvention to encode data without first using a hash function valuealgorithm or prior to using it. Thus, in some embodiments, a methodassigns a marker to each chunklet from a set of X markers to form a setof a plurality of markers, wherein X equals the number of differentcombinations of N bits within a chunklet, identical chunklets areassigned the same marker and at least one marker is smaller than thesize of a chunklet. Consequently, the set of coded data comprises saidplurality of markers. In other embodiments, the method assigns a markerto each subunit from a set of X markers to form a set of a plurality ofmarkers, wherein X equals the number of different combinations of N bitswithin a subunit, identical subunits are assigned the same marker and atleast one marker is smaller than the size of a subunit and wherein theset of coded data comprises said plurality of markers. Optionally, if asystem knows (or is designed with the premise) that for a particulartype of file not all combinations of N bits will be used, within the bitmarker table or frequency converter there may be fewer markers than allof the theoretically possible number of markers for a given chunkletsize.

During the translation process (which also may be referred to as anencoding process) the string of bits (i.e., the chunklets or subunits)that the algorithm uses as input for the bit marker table or frequencyconverter may be pre-processed. Each of these strings of bits may bedefined by a first end and a second end, and prior to assigning a markerthe method further comprises analyzing each string of bits to determineif the bit at the second end has a value of 0. If the bit at the secondend has a value of 0, the method may remove the bit at the second endand all subsequent bits that have a value of 0 and form a contiguousstring of bits with the bit at the second end, thereby forming a stringof bits of reduced size. A benefit of the pre-processing step is that asmall bit marker table or frequency converter can be used. For example,Table II could be used instead of Table I to produce the same codeddata. As persons of ordinary skill in the art will recognize, thispreprocessing can be accomplished by searching and removing 1s insteadof zeroes from the second end.

Additionally, as persons of ordinary skill in the art will recognize,Table I and Table II assign bit markers (i.e., converted bits) in amanner that is independent of the frequency of the bits in the raw data.However, as mentioned above and explained in example 3 below, one couldassign smaller markers to raw data that is expected to appear morefrequently in a document type or set of documents. This strategy takesadvantage of the fact that approximately 80% of all information iscontained within approximately the top 20% of the most frequentsubunits. In other words, the subunits that correspond to data arehighly repetitive. The converted or coded bits may be stored directly orsubjected to a hash function value algorithm as described above.

The systems of the present invention may be designed such that the hashfunction value algorithm and algorithm for conversion are either storedwithin the mediator, or the manager, or within other hardware and/orsoftware that are operably coupled to the mediator or manager. Either orboth the algorithms may also cause the file name to be stored in amediator. There are no limitations as to where the mediator isphysically located. However, preferably, it is configured to communicatewith a host or a computer that is capable of communicating with a hostthat preferably is located remote from the mediator. The mediator isalso configured to communicate, directly or indirectly (e.g., throughthe manager), with a recording medium, e.g., a non-cache medium wherethe coded set of data is stored, which optionally is remote from themediator, any manager and the host. As noted above, the mediator permitsa user who identifies the file name to retrieve the set of coded datafrom the non-cache storage medium.

In some embodiments, upon receipt of the raw data, the methods of thepresent invention may cause a confirmation of receipt to beautomatically returned to the host. In one QoS (quality of service)protocol, a data file is received through an I/O and immediately sent toL1 cache. Upon receipt, an acknowledgement is sent from L1 cache backthrough the I/O. From L1 cache, the data file may be sent to L2 cache,which transmits an acknowledgement back to L1 cache. The L2 cache mayalso send the data file to a non-cache medium (NCM) for long termstorage. The NCM may in turn send an acknowledgement back to L2 cache.

In some embodiments, the mediator may reside in or be operably coupledto a heap (dynamically allocated memory) within L1 cache. Alternatively,the mediator may reside within a card, or be part of or be operablycoupled to L2 cache.

As one of ordinary skill in the art knows, the decision to place themediator in L1 versus L2 will be impacted by factors such as thefrequency of use of the stored data. Thus, L1 cache is used to storedata that is used frequently by the system or an end user, while L2caches may be used for data that is accessed somewhat frequently.

In another QoS protocol, through the I/O, a data file is received by L1cache. The data file is transferred to both L2 cache and the NCM from L1cache. Each of L2 cache and the NCM send acknowledgments to L1 cache.Either before or after receiving acknowledgments from one or both of L2cache and the NCM, L1 cache sends an acknowledgement through the I/O.

In the various embodiments of the present invention, the host willunderstand each file to be stored at a first storage address. The firststorage address may be stored by the host in a sector map and correspondto a LUN. It may also include the start and, either implicitly orexplicitly, the end of the units, sectors or blocks that correspond to afile. The first storage address will correspond to where the hostbelieves that the file is located within a storage device or storagearea network. The host will use this first address to keep track of itsstored documents and to retrieve them. The first storage address is avirtual address i.e., it does not correspond to where the data isactually stored.

As persons of ordinary skill in the art will recognize, methods andsystems may be used in which the host generates the first storageaddress and sends it along to the systems of the present invention withSCSI commands and optionally associated sector or LBA numbers. Themediator may correlate the file name, what the host thinks of as thelocation of the file and the storage size of the file as received fromthe host, i.e., the raw data and any header or footer data, with asecond storage address, which is the actual storage address of the data,which may be converted. Alternatively, the mediator may store only thefile name, and optionally, it may not receive the first storage addressfor a file. As noted above, because storage addresses are based on alinear organization of data, they may implicitly or explicitly containthe size of the stored information.

Although the paragraphs above describe that the host will provide whatit believes to be the first storage address, the information could begenerated by another entity that either is a conduit through which thehost communicates directly or indirectly with the mediator, a modulewithin the host or operably coupled to the host, or a module within oroperably coupled to the mediator and/or manager. As persons of ordinaryskill in the art will recognize, the stored information that identifiesa location of a data file on a storage device may be referred to as apointer.

Because in many embodiments, the stored file will be smaller than theraw data file as received from the host, less storage space is neededfor it. Thus, the data needed to recreate the file can be stored in asmaller location than the host perceives is being used and than thefirst storage address suggests. The actual location in which the file isstored, may be referred to as a second storage address. Thus, for eachfile there will be a first storage address, which is where the hostbelieves that the file is stored, and a second storage address, which iswhere the coded file is actually stored.

It is possible that for a given file, which may correspond to one ormore blocks, a first storage address and a second storage address arelocated at the same block within a storage device or one or moreoverlapping set of blocks. However, preferably for at least one, atleast 50%, at least 60%, at least 70%, at least 80%, at least 90% or100% of the files there is no overlap of blocks within the first storageaddress and the second storage address. Additionally even if the hostperceives the same storage address as the mediator perceives, when datais coded the host cannot recreate the file without first decoding thedata. In some embodiments, the host is unaware of the code, and thus isnot capable of decoding the stored data.

As noted above, the mediator may comprise a first reserve set of tracks(R₁) and a second reserve set of tracks (R₂). In some embodiments, thesecond reserve set of tracks (R₂) is a copy of the first reserve set oftracks (R₁). Additionally, in some embodiments, one may use the secondreserve set of tracks (R₂) to check for errors in the first reserve setof tracks (R₁).

R₁ may be configured to function as the central point for hostinitiation. Thus, the host may select the parameters to send to R₁. Themediator may receive this information directly from the host orindirectly through the manager. R₂ is preferably never exposed to thehost. Thus, only the mediator itself or the manager can causeinformation to be stored in R₂. Each of R₁ and R₂ may for examplecontain sixteen sectors and be filled with real data such as hostmodifiers. By convention, numbering may start at 0. Thus, R₁ may forexample contain sectors (or tracks) 0-15 and R₂ may contain sectors (ortracks) 16-31. However, the mediator may be constructed so as to allowfor expansion of each of R₁ and R₂ beyond the initial size of 16 tracks.

In some embodiments, R₁ contains unique reserve sector information andpartition information. Within the partition information, one may storethe file system information.

By way of a non-limiting example and as persons of ordinary skill in theart are aware, when formatting a volume with an NFTS file system, onecreates metadata files such as $MFT (Master File Table), $Bitmap, $LogFile and others. This metadata contains information about all of thefiles and folders on an NFTS volume. The first information on an NTFSvolume may be a Partition Boot Sector ($Boot metadata file), and belocated at sector 0. This file may describe the basic NTFS volumeinformation and a location of the main metadata file $MFT.

The formatting program allocates the first 16 sectors for the $Bootmetadata file. The first sector is a boot sector with a bootstrap code,and the following 15 sectors are the boot sector's IPL (initial programloader).

In addition to the tracks of R₁ and R₂, the mediator may storeadditional metadata. This metadata may for example correspond toinformation that allows the execution of thin provisioning strategies,which correspond to visualization technology that allows a device togive the appearance of having more physical resources than are actuallyavailable, and it may, for example, be contained in the eight tracksafter R₂, which would be tracks 32-39. The metadata may also provide forfeatures such as cubelet QoS, VM and WORM.

Finally, the mediator may also comprise a bit field. The bit fieldcontains the information that indicates where the data is physicallystored within a storage medium and if the metadata is located in tracks32-39, the sector number of the bit field begins at track 40. It iswithin the bit field of the mediator that correlation between the filename of the host and the location of the data is stored. Thus, it maycomprise, consist essentially of or consist of a sector map. Thisinformation from the bit table component of the mediator may be used todetermine the actual space saving on any device. For example, thepercentage of space saved=1−[(space actually used)/(space as mapped byhost)]. In some embodiments, the space saved is at least 50%, at least60%, at least 70% or at least 80%. These savings may be per file oraveraged over all files on a device.

As a matter of practice, preferably the mediator is not located on thedisk or recording medium on which the coded data is stored.Additionally, preferably the mediator requires only about 0.1-0.2% ofthe total memory of the corresponding disk or recording medium.

In addition to providing economic value from saving of space, variousembodiments of the present invention open the door for increasedefficiencies when looking to protect the integrity of data. Accordingly,various embodiments of the present invention provide new and non-obvioustechnologies for backing-up data.

In certain embodiments, one may use two mediators to facilitatebacking-up data. For example, in a first mediator one may correlate adata file that is stored in a first sector or first sector cluster on afirst recording medium with a file name. As described above, the firstmediator is configured to permit a user or entity that identifies thefile name to retrieve the data file from the recording medium.

A data protection protocol may be executed that generates a secondmediator. The second mediator will be an exact copy of the firstmediator at a time T1. Thus, at T1, both the first mediator and thesecond mediator will point to the same sectors or sector clusters (andlocations therein) on the first recording medium.

After time T1, for example at T2, the host may seek to update a filethat is stored in a given location on a given sector or sector cluster.The host will not change the data stored at the first storage address.However, rather than causing the information on the sector or sectorcluster to be written over, the first mediator may cause the newinformation to be written to a third storage address that corresponds toa location in a different sector or sector cluster and correlate thefile name and the first storage address with this new storage address.

Thus, the first mediator will point to a new sector or sector clustereven though the host believes that the information is being overwrittenat a particular storage address. Accordingly, the host will not need toupdate its address for the sector cluster.

Additionally, the second mediator will not be updated, and it willcontinue to correlate the file name with the first location at which thefile was stored. This use of the two mediators will permit one toprovide a snapshot of the data as it existed at T1, without causing thehost to need to update its file system to indicate that the file as itexisted both at T1 and at T2 are being stored. Thus, the snapshot locksall data files that are stored at time T1 and prevents anyone fromdeleting or writing over those physical files. However, if the hostwishes to revise those files, it can work under the impression that itis doing so, when in fact new files are stored. This method is describedin connection with sectors and sector clusters. However, it will alsowork with non-cache media that are not arranged in sectors or sectorclusters. For example, they may be organized by LBAs in LUNs.

As suggested above, this method may be implemented by a system thatcomprises a first mediator, a second mediator and a non-cache storagemedium. Each of the first mediator, the second mediator and therecording medium may be stored on or be formed from separate devicesthat comprise, consist essentially of or consist of non-transitorymedia. The afore-described system recites the use of different sectorsof the same recording medium but could also be used by writing todifferent sectors of different recording media. Additionally, within thesystem the mediators and the recording media are operably coupled to oneanother and optionally to one or more computers or CPUs that storeinstructions to cause them to carry out their intended functions and tocommunicate through one or more portals over a network to one or morehosts. Still further, although this embodiment is described inconnection with the use of two mediators, one could implement the systemusing two sections of the same mediator rather than two separatemediators.

The system may further be configured with a locking module. The lockingmodule may prevent revision, overwriting or deletion at one or moreblocks that have been written as of a certain time. The locking modulemay also be designed to allow for the writing of new blocks and revisionof those new blocks that have not been locked. Thus, the locking modulemay be configured to permit a host, a user or a system administrator toselect certain blocks that have been written as of a certain time or toselect all blocks that have been written as of a certain time not to beoverwritten.

Furthermore, there may be a selection module that by default sends allrequests for retrieval of files and revision, overwriting or deletionthrough the first mediator. The selection module may also be configuredto allow recovery of what a host may believe are older versions of oneor more files as of the times at which the locking technology wasapplied. Optionally, access to the entire selection module may berestricted to persons who have authorization, e.g. a systemadministrator.

The aforementioned system for backing-up data is described in thecontext of two mediators. However, more than two mediators could be usedto capture a history of stored files or versions of files.

In some embodiments the system may contain a SAN indexer. The SANindexer may check what is in R₁ and R₂, and extract that information.This information can be put into a database that may readily be searchedby, for example, text searching.

According to another method for backing up data, a clone of thenon-cache media may be made. In this method, in a first medium, onecorrelates a plurality of file names with a plurality of locations ofdata that are stored on a non-cache storage medium. The first mediatoris configured to permit a user who identifies a specific file name toretrieve a data file from the first non-cache storage medium thatcorresponds to the specific file name. Part or the entire specific filemay be stored in a first sector or sector cluster.

One may make a copy of the plurality of data files (or all data files ofa first non-cache storage medium) to a second non-cache storage mediumand a second mediator. The second mediator is a copy of the firstmediator at time T1 and is operably coupled to the second non-cachestorage medium. At time T2, which is after T1, the system may saverevisions to a data file that is stored in said first sector or sectorcluster on the first non-cache storage medium. However, no changes wouldbe made to the corresponding location on the second non-cache medium. Asa user requests a file after T2, he or she would go through the firstmediator and retrieve the most recent stored version of the file.However, the system administrator would have access to an earlierversion, which would be stored on the second non-cache medium and couldretrieve it by going through the second mediator.

This method may be implemented by a system that comprises a firstmediator, a second mediator, a first non-cache storage medium and asecond non-cache storage medium. Each of the first mediator, the secondmediator and the first and second recording media for storing data filesmay be stored on separate devices that comprise, consist essentially ofor consist of non-transitory media. Additionally, the first mediatorcorrelates a file name that is derived from a host with a first cubeletof the first recording medium and the second mediator correlates thesame file name with a second cubelet on the second recording medium. Insome embodiments, the most recent file, which is stored in the firstnon-cache medium, has the same LUN that the legacy file has within thesecond non-cache medium.

Retrieval of the data as stored may be through processes andtechnologies that are now known or that come to be known and that aperson of ordinary skill in the art would appreciate as being of use inconnection with the present invention. Optionally, a manager coordinatesstorage and retrieval of files.

After the data is retrieved from a recording medium, if the data hasbeen converted, then one translates the plurality of markers (or datathat has been converted through the frequency converter) into bits thatmay be used to form chunklets or hash values to be converted back intothe I/O stream or the fragmented units of the I/O stream. The markersmay be stored such that each marker corresponds to a chunklet or eachmarker corresponds to a subunit and a plurality of subunits may becombined to form a chunklet. In the stored format, the markers arearranged in an order that permits recreation of bits within chunklets(or hash values) and recreation of the order of chunklets (or hashvalues), in a manner that allows for recreation of the desired documentor file.

If the data is converted, then in order to translate the markers intochunklets, one may access a bit marker table or a frequency converter.Within the bit marker table or frequency converter, there may be aunique marker that is associated with each unique string of bits orwithin each unique string of bits within the file. If the table isorganized in a format similar to Table II, after translation, zeroes maybe added in order to have each subunit and chunklet be the same size.When decoding, one uses the bit maker table or frequency converter inthe opposite way that one would use this for coding. Optionally, insteadof using the same table and reversing the input and output, one coulduse separate tables.

After the chunklets are formed, in some embodiments, one will have anoutput that corresponds to binary data from which a file can bereconstituted. In other embodiments, the converted data corresponds tohash values, and the hash values must first be used to create fragmentedunits of the I/O stream or the I/O stream itself.

Optionally, one may associate the file with a file type. The file typewill direct the recipient of the data to know which operating systemshould be used to open it. In some embodiments, the association with afile type is done at the initiator or client or host.

In order to illustrate the various embodiments further and to providecontext, reference is made below to specific hardware that one may use,which may be combined to form a system to implement the methods of thepresent invention. The hardware may be configured to allow for the abovedescribed process to be automated and controlled by one or moreprocessors.

In some embodiments, a host may generate documents and computer files inany manner at a first location. The documents will be generated by thehost's operating system and organized for storage by the host's filesystem. The host's operating system may locally store in its memory, thefile name. The present invention is not limited by the type of operatingsystem or file system that a host uses. By way of a non-limitingexample, the host may comprise a computer or set of computers within anetwork having one or more of the following hardware components: memory,storage, an input device, an output device, a graphic user interface,one or more communication portals and a central processing unit.

At that first location a SAP executes a protocol for storing the datathat correlates to documents or files. The SAP formats the data into I/Ostreams or chunklets that are for example 4K in size.

The data may be sent over a SAN to a computer that has one or moremodules or to a computer or set of computers that are configured toreceive the data. This receiving computer may comprise one or more ofthe following hardware components: memory, storage, an input device, anoutput device, a graphic user interface, a central processing unit andone or more communication portals that are configured to permit thecommunication of information with one or more hosts and one or morestorage devices locally and/or over a network.

Additionally, there may be a computer program product that stores anexecutable computer code on hardware, software or a combination ofhardware and software. The computer program product may be divided intoor able to communicate with one or more modules that are configured tocarry out the methods of the present invention and may be stored in oneor more non-transitory media.

In some embodiments, there may be a level 1 (L1) cache and a level 2(L2) cache. In the present invention, by way of an example, the data maybe sent over a SAN to a cache and the data may be sent to the cacheprior to accessing a hash function algorithm, prior to consulting a bitmarker table, prior to consulting a frequency converter, and prior totruncating bits, and/or after consulting a hash function algorithm,after consulting a bit marker table, after consulting a frequencyconverter, and after truncating bits.

Assuming that the sector size is 512B, for each chunklet that is 4K insize, the host will expect 8 sectors of storage to be used.

In another embodiment, the present invention provides a data storage andretrieval system. The system comprises a non-cache data storage medium,a mediator and a manager. Communication among these elements andoptionally the initiator may be over a network that is wired, wirelessor a combination thereof.

The non-cache data storage medium may for example comprise, consistessentially of or consist of one or more disks or solid state drives.When in use, the non-cache data storage medium may store files with aspace savings of at least 50%, at least 60%, at least 70% or at least80%.

The mediator may comprise, consist essentially of or consist of foursets of tracks: a first set of tracks, a second set of tracks, a thirdset of tracks and a fourth set of tracks. The mediator is preferablystored on a non-transitory medium and is located remotely from thenon-cache data storage medium. Thus, the mediator and the non-cache datastorage medium are preferably not part of the same device.

The system may also contain a manager. The manager may provide thecontrol of the receipt, processing storage and retrieval andtransmission of data through the mediator. Thus, preferably, it isoperably coupled to the host and the mediator and optionally operablycoupled to the non-cache data storage medium. Furthermore, in someembodiments it is located remotely from each of the mediator, thenon-cache medium and the host.

The manager may be configured to carry out one or more of the followingfeatures: (a) to store data comprising one or more of file systeminformation, bootability information and partition information in thefirst set of tracks; (b) to store metadata in the third set of tracks;(c) to store one or more files on the non-cache medium, wherein the oneor more files are stored on the non-cache medium without any of filesystem information, bootability information or partition information(thus in some embodiments, only raw data is on the non-cache medium);(d) to store in the fourth set of tracks the location of each file inthe non-cache medium; and (e) to store a correlation of the location ofeach file in the non-cache medium with a host name for a file.Preferably, the correlation of the location of each file in thenon-cache medium with a host name for a file is stored in the fourth setof tracks, which corresponds to a bit field. Additionally, the managermay comprise or be operably coupled to a computer program product thatcontains executable instructions that are capable of executing one ormore of the methods of the present invention.

For purposes of further illustration, reference may be made to thefigures. FIG. 1 shows a system 100 with a mediator 10 that contains R₁40 and R₂ 50, as well as a space for a bit field 60 and metadata files30. The representation of the mediator is for illustrative purposes onlyand places no limitations on the structure of the mediator ororganization within it. Also shown is a non-cache medium (NCM) 20. Thenon-cache medium is shown in the proximity of the mediator, but they areseparate structures.

FIG. 2 shows another system 200. In this system, the initiator (I^(n))270 transmits chunklets to a cache manager 230, which optionallyarranges for coding of data files and transmits them to the mediator210. Examples of hosts include but are not limited to computers orcomputer networks that run Microsoft Windows Server and Desktop, AppleOS X, Linux RHEL and SUSE Oracle Solaris, IBM AIX, HP UX and VM ESX andESXi. The information, corresponding to data files, is initially sent toR₁ 240, which previously was populated with parameters that theinitiator defined. The mediator may itself translate the informationthrough use of a bit marker table or a frequency converter (not shown)or it may communicate with a remote encoder (which also may be referredto as a remote converter and is not shown), and the mediator will storewithin R₁ as well as within R₂ 250 copies of a file name that isreceived from the host. After the data has been converted, and a smallersize file has been created, within a sector map of the bit field 260, isrecorded a location that the file will be stored in the disk 220. Thecoded data will be stored at location 285. Prior to or instead of codingdata, a hash value algorithm may be accessed and the checksums may beused for storage or conversion.

FIG. 3 shows another system 300 that is a variation of the embodiment ofthe system of FIG. 2 and that provides for back-up of storage. In thissystem the initiator 370 transmits chunklets to the cache manager 330,which forwards information to the mediator 310 that contains data torevise the same file that was sent for FIG. 2. Either prior to receiptof the revised file or after receipt of it, but before storage of it inthe non-cache media, a second mediator 380 is created from the firstmediator 310. The second mediator is an exact copy of the first mediatorat the time that it was created and for the file name, at that time,points to the same sector (or sector cluster) 385 within the non-cachemedium 320.

The first revised file is received at R₁ 340 of the first mediator. Thefirst mediator will again either translate the information through useof a bit marker table or a frequency converter (not shown) orcommunicate with a remote encoder. The mediator will continue to storewithin R₁ as well as within R₂ 350 copies of the file name that isreceived from the host. After the data has been converted, and a smallersize file has been created, within a sector map of the bit field 360 ofthe first mediator, is recorded a location that the file will be storedin the disk 320. However, the revised file will be stored in a differentsector 395. Thus, the changes to the first mediator will not be made tothe second mediator.

The host is by default in communication with the first mediator. Thus,when it wants to recall the file from storage, the first mediator willcall back the data from sector 395. Should the host or a systemadministrator wish to obtain a previous version of the data, it couldsubmit the file name to the second mediator, which would look to sector385.

According to any of the methods of the present invention, any data thatis stored in a converted form is capable of being retrieved and decodedbefore returning it to a host. Through the use of one or more algorithmsthat permit the retrieval of the converted data, the accessing of thereference table or frequency converter described above and theconversion back into a uniform string of bits and chunklets, files canbe recreated for hosts. By way of a non-limiting example, the data maybe converted and stored in a format that contains an indication whereone marker ends e.g., use of unique strings of bits.

As persons of ordinary skill in the art will recognize, certainembodiments of the present invention are described in connection withone or two non-cache media. However, an initiator may be associated witha plurality of mediators and a plurality of non-cache media.

Any of the features of the various embodiments described in thisspecification can be used in conjunction with features described inconnection with any other embodiments disclosed unless otherwisespecified. Thus, features described in connection with the various orspecific embodiments are not to be construed as not suitable inconnection with other embodiments disclosed herein unless suchexclusivity is explicitly stated or implicit from context.

EXAMPLES Example 1 Bit Marker Table (Prophetic)

Within a reference locator table each unique marker is identified ascorresponding to unique strings of bits. The table may be stored in anyformat that is commonly known or that comes to be known for storingtables and that permits a computer algorithm to obtain an output that isassigned to each input.

Table I below provides an example of excerpts from a bit marker tablewhere the subunits are 8 bits long.

TABLE I Bit Marker (as stored) Subunit = 8 bits (input) 0101 000000011011 00000010 1100 00000011 1000 00000100 1010 00000101 1111110111111101

By way of example and using the subunits identified in Table I, if theinput were 00000101 00000100 00000101 00000101 00000001, the outputwould be: 1010 1000 1010 1010 0101. When the bit marker output issmaller than the subunit input, it will take up less space on a storagemedium, and thereby conserve both storage space and the time necessaryto store the bits.

As a person of ordinary skill in the art will recognize, in a given bitmarker table such as that excerpted to produce Table I, if allcombination of bits are to be used there will need to be 2^(N) entries,wherein N corresponds to the number of bits within a subunit. When thereare 8 bits, there are 256 entries needed. When there are 16 bits in asubunit one needs 2¹⁶ entries, which equals 65,536 entries. When thereare 32 bits in a subunit, one needs 2³² entries, which equals4,294,967,296 entries. If one knows that certain strings of bits willnot be used in files, then the table may allocate markers starting withthe smallest ones.

Example 2 Bit Marker Table for Pre-Processed Subunits (Prophetic)

Because as the subunit size gets larger the table becomes morecumbersome, in some embodiments, the table may be configured such thatall zeroes from one end of the subunit column are missing and prior toaccessing the table, all zeroes from that end of each subunit areremoved. Thus, rather than a table from which Table I is excerpted, atable from which Table II is excerpted could be configured.

TABLE II Bit Marker (output) Pre-processed Subunit 0101 00000001 10110000001 1100 00000011 1000 000001 1010 00000101 11111101 11111101

As one can see, in the second and fourth lines, after the subunits werepre-processed, they had fewer than eight bits. However, the actualsubunits in the raw data received from the host all had eight bits.Because the system in which the methods are implemented can be designedto understand that the absence of a digit implies a zero and allabsences of digits are at the same end of any truncated subunits, onecan use a table that takes up less space and that retains the ability toassign unique markers to unique subunits. Thus, the methods permit thesystem to interpret 00000001 (seven zeroes and a one) and 0000001 (sixzeroes and a one) as different.

In order to implement this method, one may deem each subunit (or eachchunklet if subunits are not used) to have a first end and a second end.The first end can be either the right side of the string of bits or theleft side, and the second end would be the opposite side. For purposesof illustration, one may think of the first end as being the leftmostdigit and the second end as being the rightmost digit. Under this methodone then analyzes one or more bits within each subunit of each chunkletto determine if the bit at the second end has a value 0. This step maybe referred to as preprocessing and the subunits after they arepreprocessed appear in the right column of Table II. If the bit at thesecond end has a value 0, the method may remove the bit at the secondend and all bits that have the value 0 and form a contiguous string ofbits with that bit, thereby forming a revised subunit (pre-processedsubunit in the table) for any subunit that originally had a 0 at thesecond end.

One may use a computer algorithm that reviews each subunit to determinewhether at the second end there is a 0 and if so removes the 0 to formthe pre-processed subunit, which also may be referred to as a revisedsubunit with a revised second end at a position that was adjacent to thesecond end of the subunit. Next, the algorithm reviews the revisedsubunit to determine whether at its now revised second end there is a 0and if so removing the 0 to form a further revised second end. In thismethod, the revised second end would be the location that was previouslyadjacent to the bit at the second end. Any further revised second endwould have been two or more places away from the second end of thesubunit. Thus, the term “revised” means a shortened or truncated secondend. The algorithm may repeat this method for the revised subunit untila shortened chunklet is generated that has a 1 at its second end.

Example 3 Frequency Exchange (Prophetic)

Based on empirical analysis, one can determine the frequency of eachsubunit within a type of document or a set of documents received from aparticular host or from within a set of documents that have beenreceived within a given timeframe, e.g., the past year or past twoyears. With this information, rather than look to a table as illustratedin Table I or Table II in which the subunits are organized in numericalorder, one could look to a frequency converter in which the smaller bitmarkers are associated with subunits that are predicted most likely toappear within a file, within a type of document or within a set ofdocuments as received from a particular host. Thus, within the frequencyconverter, the markers are a plurality of different sizes and markers ofa smaller size are correlated with higher frequency subunits.

TABLE III Frequency Converter Bit Marker (output) Frequency Subunit = 8bits (input) 0101 16% 00000001 1000 15% 00000010 11011 10% 0000001110011101 0.00001%    00000100 10111110 0.00001%    00000101 1100 15%11111101

Table III is an example of an excerpt from a frequency converter thatuses the same subunits as Table I. However, one will note that the bitmarkers are not assigned in sequence, and instead larger bit markers areassigned to lower frequency subunits. As the table illustrates, themarker that is assigned to subunit 00000011 is twenty five percentlarger than that assigned to subunit 00000001, and for subunit 11111101,despite being of high numerical value, it receives a smaller bit markerbecause it appears more frequently in the types of files received fromthe particular host. Thus, if one used Table I and the subunit 11111101appears in 10,000 places, it would correspond to 111,111,010,000 bits.However, if one used Table III, only 11,000,000 bits would need to beused for storage purposes for the same information. Although not shownin this method, the subunits could be preprocessed to remove zeroes fromone end or the other, and the table could be designed to contain thecorrelating truncated subunits.

As noted above, frequency converters can be generated based on analysesof a set of files that are deemed to be representative of data that islikely to be received from one or more hosts. In some embodiments, thealgorithm that processes the information could perform its own qualitycontrol and compare the actual frequencies of subunits for documentsfrom a given time period with those on which the allocation of marker inthe frequency converter are based. Using statistical analyses it maythen determine if for future uses a new table should be created thatreallocates how the markers are associated with the subunits. As aperson of ordinary skill in the art will recognize, Table III is asimplified excerpt of a frequency converter. However, in practice onemay choose a hexadecimal system in order to obtain the correlations.Additionally, the recitation of the frequencies on which the table isbased is included for the convenience of the reader, and they need notbe included in the table as accessed by the various embodiments of thepresent invention.

Example 4 Allocation of Space in a Mediator (Prophetic)

In a hypothetical recording medium that is 1 MB in size, a person ofordinary skill in the art may map the sectors as follows:

The 1 MB recording medium has 1,024,000 Bytes, which corresponds to 250sectors. (1,024,000/4096=250). The geometry of the recording medium maybe summarized as follows: Volume=(c*h*spt*ss), wherein

c (number of cylinder)=7;

h (number of heads)=255;

spt (sectors per track)=63; and

ss (sector size in bytes)=4096.

Within the mediator, the sectors may be allocated as follows:

TABLE IV Address Actual Non-cache-media LBA  0-15 mediator <<Reserved1>> “Boot Sector 0” +15 16-31 mediator location <<Reserved 2>>Sys_Internal Only 32-35 mediator_Metadata 36 Map Data“LBA-nnnnnnnnnnnnn” 37 Map Data “LBA-nnnnnnnnnnnnn” . . . Map Data“LBA-nnnnnnnnnnnnn” 250  Map Data “LBA-nnnnnnnnnnnnn”

Example 5 Space Saving

A system of the present invention received 42.5 million blocks of datain I/O streams of 4096 Bytes. It applied the MD5 hash algorithm togenerate 7.8 million blocks of hash value that corresponded to the 42.5million blocks in the I/O stream.

This translated into use of only 18.5% of the space that would have beenneeded to store the original 42.5 million blocks. A conflict module wasapplied, and it verified that no conflicts existed, i.e., no duplicationof hash values, were generated for different blocks.

1.-19. (canceled)
 20. A method for storing data on a non-cache recordingmedium comprising: i. receiving an I/O stream of N Bytes; ii.fragmenting the N Bytes into fragmented units of X Bytes; iii. applyinga cryptographic hash function to each fragmented unit of X Bytes to forma generated hash function value for each fragmented unit of X Bytes; iv.encoding each hash function value for storage through use of a bitmarker table to generate a bit marker for each hash function value forstorage and storing each bit marker in the non-cache recording medium;v. accessing a correlation file, wherein the correlation file associatesa stored hash function value of Y bits with each of a plurality ofstored sequences of X Bytes and (a) if the generated hash function valuefor a fragmented unit of X Bytes is in the correlation file, using thestored hash function value of Y bits for storage on a non-cacherecording medium; and (b) if the generated hash function value for thefragmented unit of X Bytes is not in the correlation file, then storingthe generated hash function value of Y bits with the fragmented unit ofX Bytes in the correlation file and using the generated hash functionvalue for storage on the non-cache recording medium.
 21. The methodaccording to claim 20, wherein in (iii) the cryptographic hash functionis the MD5 Message-Digest Algorithm.
 22. The method according to claim20 further comprising storing a plurality of hash function values on thenon-cache recording medium.
 23. The method according to claim 20 furthercomprising encoding each hash function value for storage through use ofa frequency of occurrence converter, wherein for each hash functionvalue for storage, a converted string of bits is generated, wherein forat least two different strings of Y bits, the converted strings of bitsthat are output are of different lengths.
 24. The method according toclaim 23, wherein for every plurality of converted strings of bits thatare of different sizes there is a first converted string of bits that isA bits long and a second converted string of bits that is B bits long,wherein A<B, and the identity of the A bits of the first convertedstring of bits is not the same as the identity of the first A bits ofthe second converted string of bits.
 25. The method according to claim20, wherein the I/O stream is received from a host and said methodfurther comprises acknowledging receipt of the I/O stream to the host.26. The method according to claim 25, wherein the host records the I/Ostream as being stored at a first storage address and the convertedstring of bits are stored in the non-cache recording medium at a secondstorage address, wherein the first storage address is not the same asthe second storage address.
 27. The method according to claim 26,wherein the first storage address corresponds to a first file size andthe second storage address corresponds to a second file size, whereinthe first file size is at least four times as large as the second filesize.
 28. The method according to claim 26, further comprising storingthe second storage address on a mediator, wherein the mediator is anontransitory storage medium.
 29. The method according to claim 28,wherein the second storage address is stored in a bit field.
 30. Themethod according to claim 29, wherein the mediator comprises: i. a firstset of tracks, wherein the first set of tracks comprise file systeminformation, bootability information and partition information; ii. asecond set of tracks, wherein the second set of tracks comprise a copyof the first set of tracks; iii. a third set of tracks, wherein thethird set of tracks comprises metadata other than file systeminformation, bootability information and partition information; and iv.a fourth set of tracks, wherein the fourth set of tracks comprise thebit field.
 31. The method according to claim 28, wherein the mediator isnot located on a storage medium on which the bit markers are stored. 32.A method for storing data on a non-cache recording medium comprising: i.receiving an I/O stream of N Bytes; ii. fragmenting the N Bytes intofragmented units of X Bytes; iii. associating a cryptographic hashfunction value with each fragmented unit of X Bytes, wherein saidassociating comprises accessing a correlation file, wherein thecorrelation file associates a stored hash function value of Y bits witheach of a plurality of stored sequences of X Bytes and (a) if thesequence of fragmented unit of X Bytes is in the correlation file usingthe stored hash function value of Y bits for storage on a non-cacherecording medium; and (b) if the sequence of fragmented unit of X Bytesis not in the correlation file, then storing a new hash function valueof Y bits with the fragmented unit of X Bytes in the correlation fileand using the new hash function value of Y bits for storage on thenon-cache recording medium.
 33. A method for storing data on a non-cacherecording medium comprising: i. receiving an I/O stream of N Bytes; ii.applying a cryptographic hash function value algorithm to each unit of NBytes to form a generated hash function value for each unit of N Bytes;iii. accessing a correlation file, wherein the correlation fileassociates a stored hash function value of Y bits with each of aplurality of stored sequences of N Bytes and (a) if the generated hashfunction value for the unit of N Bytes is in the correlation file, usingthe stored hash function value of Y bits for storage on a non-cacherecording medium; and (b) if the generated hash function value for theunit of N Bytes is not in the correlation file, then storing thegenerated hash function value of Y bits with the unit of N Bytes in thecorrelation file and using the generated hash function value for storageon the non-cache recording medium.